{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "mount_file_id": "176FD0oaK-tAn_U5nmqtNnBdHWGre6AXc",
      "authorship_tag": "ABX9TyNCQrTgEMSEIQdws5Rd1mtQ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chenyu313/Colaboratory_note/blob/main/colab%E7%9A%84%E4%B8%80%E4%BA%9B%E5%B0%8F%E6%8A%80%E5%B7%A7.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 一、初始设置\n",
        "## 1.占用显存\n",
        "首先输入命令，占用1个G的显存，让谷歌认为你没有滥用显存。"
      ],
      "metadata": {
        "id": "g2i6_N7YbfKc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "a = torch.Tensor([1000, 1000, 1000]).cuda()  # 大约占用 1.1G 显存"
      ],
      "metadata": {
        "id": "wgbCkJRacGzV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.查看分配给自己的显卡\n"
      ],
      "metadata": {
        "id": "XBDidHVQcZLV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YI6NHvtXcdjT",
        "outputId": "1c9b6351-c30b-438a-9b51-df6d4d7f40dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fri Feb 24 03:22:58 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   62C    P0    30W /  70W |    570MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|    0   N/A  N/A      2391      C                                     567MiB |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 二、训练部分\n",
        "## 1.挂载google drive\n"
      ],
      "metadata": {
        "id": "bBCM0YxsfKnu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H185YubXfZB7",
        "outputId": "d3d7f4c8-baec-4eca-fa55-d2cb825e730b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.更改工作目录\n",
        "可以将工作路径更改  \n",
        "import os  \n",
        "os.chdir(\"更改后的路径\")   "
      ],
      "metadata": {
        "id": "mTcYFRsJf73m"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3.移动训练数据\n",
        "训练数据最好复制一份到服务器里，训练数据最好复制一份到服务器里。 \n",
        "\n",
        "\n",
        "```\n",
        "!cp /content/drive/具体训练数据地址 /content   \n",
        "!unzip -d /content/centernet\n",
        "```\n",
        "如下： \n",
        "\n",
        "```\n",
        "!cp /content/drive/Shareddrives/deeplearning_data/centernet-pytorch.zip /content \n",
        "!unzip -d /content/centernet\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "  "
      ],
      "metadata": {
        "id": "6HK6XmwZgO3N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.开始训练"
      ],
      "metadata": {
        "id": "l2G-kELmhzN9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python train.py"
      ],
      "metadata": {
        "id": "2H2IGv89Hhbs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5.防止断连\n",
        "* 方法一：每60分钟自动运行代码刷新，解除90分钟断开限制. 使用方法：colab页面按F12或者 Ctrl+Shift+I (mac按 Option+Command+I) 在console（控制台） 输入以下代码并回车. 复制以下代码粘贴在浏览器console！！不要关闭浏览器以免失效\n",
        "\n",
        "\n",
        "```\n",
        "function ConnectButton(){\n",
        "    console.log(\"Connect pushed\"); \n",
        "    document.querySelector(\"#connect\").click() \n",
        "}\n",
        "setInterval(ConnectButton,60000);\n",
        "```\n",
        "* 方法二：安装浏览器插件Colab Alive\n",
        "\n",
        "![image.png]()\n"
      ],
      "metadata": {
        "id": "M2BWxI9gH2VV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6.如何加断点存储模型参数\n",
        "每个epoch都用torch.save()存储一下模型参数。然后重新训练时加载模型选择相应的pth模型参数即可。  \n",
        "由于Colab随时有可能断开连接，在Colab上训练模型的代码必须要有可恢复性（能载入上一次训练的结果）。两个分别实现保存和加载checkpoint的函数附在下方（基于pytorch）。\n",
        "\n",
        "\n",
        "```\n",
        "def save_checkpoint(path: Text,\n",
        "          epoch: int,\n",
        "          modules: Union[nn.Module, Sequence[nn.Module]],\n",
        "          optimizers: Union[opt.Optimizer, Sequence[opt.Optimizer]],\n",
        "          safe_replacement: bool = True):\n",
        "\n",
        "    \"\"\"\n",
        "    保存当前训练状态的检查点，以便可以恢复。\n",
        "\n",
        "    这个检查点函数假设没有用于自动混合精度的学习率调度器或梯度缩放器。\n",
        "    :param path:\n",
        "        检查点文件路径\n",
        "    :param epoch:\n",
        "        当前(已完成)epoch\n",
        "    :param modules:\n",
        "        包含模型或模块对象列表的模块\n",
        "    :param optimizers:\n",
        "        优化器或优化器列表\n",
        "    :param safe_replacement:\n",
        "        保留旧的检查点，直到新的检查点完成\n",
        "    :return:\n",
        "    \"\"\"\n",
        "\n",
        "#此函数可以被称为\n",
        "#save_checkpoint('/my/checkpoint/path.pth', my_epoch, my_module, my_opt)\n",
        "#或\n",
        "#save_checkpoint('/my/checkpoint/path.pth', my_epoch, [my_module1, my_module2], [my_opt1, my_opt2])\n",
        "\n",
        "    if isinstance(modules, nn.Module):\n",
        "        modules = [modules]\n",
        "    if isinstance(optimizers, opt.Optimizer):\n",
        "        optimizers = [optimizers]\n",
        "\n",
        "#待保存的数据字典\n",
        "    data = {\n",
        "        'epoch': epoch,\n",
        "        # Current time (UNIX timestamp)\n",
        "        'time': time.time(),\n",
        "        # State dict for all the modules\n",
        "        'modules': [m.state_dict() for m in modules],\n",
        "        # State dict for all the optimizers\n",
        "        'optimizers': [o.state_dict() for o in optimizers]\n",
        "    }\n",
        "#安全更换旧检查点\n",
        "    temp_file = None\n",
        "    if os.path.exists(path) and safe_replacement:\n",
        "        # There's an old checkpoint. Rename it!\n",
        "        temp_file = path + '.old'\n",
        "        os.rename(path, temp_file)\n",
        "#保存新的检查点\n",
        "    with open(path, 'wb') as fp:\n",
        "        torch.save(data, fp)\n",
        "        # Flush and sync the FS\n",
        "        fp.flush()\n",
        "        os.fsync(fp.fileno())\n",
        "#删除久的检查点\n",
        "    if temp_file is not None:\n",
        "        os.unlink(path + '.old')\n",
        "\n",
        "def load_checkpoint(path: Text,\n",
        "          default_epoch: int,\n",
        "          modules: Union[nn.Module, Sequence[nn.Module]],\n",
        "          optimizers: Union[opt.Optimizer, Sequence[opt.Optimizer]],\n",
        "          verbose: bool = True):\n",
        "    \"\"\"\n",
        "    尝试加载一个检查点以恢复训练。\n",
        "    :param path:\n",
        "        检查点文件的路径\n",
        "    :param default_epoch:\n",
        "        “epoch”的初始值(以防没有快照)\n",
        "    :param modules:\n",
        "        包含模型或模块对象列表的模块。它们被假定停留在同一设备上\n",
        "    :param optimizers:\n",
        "        优化器或优化器列表\n",
        "    :param verbose:\n",
        "        详细模式\n",
        "    :return:\n",
        "        Next epoch\n",
        "    \"\"\"\n",
        "    if isinstance(modules, nn.Module):\n",
        "        modules = [modules]\n",
        "    if isinstance(optimizers, opt.Optimizer):\n",
        "        optimizers = [optimizers]\n",
        "\n",
        "    # 如果有检查点的话\n",
        "    if os.path.exists(path):\n",
        "        # 加载数据\n",
        "        data = torch.load(path, map_location=next(modules[0].parameters()).device)\n",
        "\n",
        "        # 通知用户我们正在加载检查点\n",
        "        if verbose:\n",
        "            print(f\"Loaded checkpoint saved at {datetime.fromtimestamp(data['time']).strftime('%Y-%m-%d %H:%M:%S')}. \"\n",
        "                  f\"Resuming from epoch {data['epoch']}\")\n",
        "\n",
        "        # 所有模块的加载状态\n",
        "        for i, m in enumerate(modules):\n",
        "            modules[i].load_state_dict(data['modules'][i])\n",
        "\n",
        "        # 所有优化器的加载状态\n",
        "        for i, o in enumerate(optimizers):\n",
        "            optimizers[i].load_state_dict(data['optimizers'][i])\n",
        "\n",
        "        # Next epoch\n",
        "        return data['epoch'] + 1\n",
        "    else:\n",
        "        return default_epoch\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "moKwiz48JoyU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "在主程序train.py正式训练前，添加下面的语句：\n",
        "\n",
        "\n",
        "```\n",
        "if args.resume: # args.resume是命令行输入的参数，用于指示要不要加载上次训练的结果\n",
        "    first_epoch = load_checkpoint(checkpoint_path, first_epoch, net_list, optims_list)\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "o41LM-8Eq1xM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "在每个epoch训练结束后，保存checkpoint：\n",
        "\n",
        "\n",
        "```\n",
        "# Save checkpoint\n",
        " save_checkpoint(checkpoint_path, epoch, net_list, optims_list)\n",
        "```\n",
        "net_list是需要保存的网络列表，optims_list是需要保存的优化器列表\n",
        "\n",
        "这里没有记录scheduler的列表，如果代码里用到了scheduler，那也要保存scheduler的列表。\n"
      ],
      "metadata": {
        "id": "lo-0T9P4rZR8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 参考文章\n",
        "https://zhuanlan.zhihu.com/p/387509768  \n",
        "https://zhuanlan.zhihu.com/p/527663163"
      ],
      "metadata": {
        "id": "S2ySBtO-rmE7"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Z0F8QK5apFhF"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}